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(54) Title: ANALYSIS OF CHEMICAL DATA FROM IMAGES 
(57) Abstract 

Apparatus and methods for extracting and analyzing 
combinatorial chemical data from images (110). The inven- 
tion receives a stream of data (120), identifies a plurality of 
regions of interest and determines a series of values for one 
or more regions of interest. The library includes a plurality 
of members. The data stream includes a series of images of 
the library. Each region of interest corresponds to a location 
in each of the images and to a location in the library. The 
series of values for each region includes a value for each of 
the images. The stream of data comprises a series of im- 
ages generated at a frequency of greater than about 1,6, 12 
or 30 frames per second. The series of images are captured 
from a beginning of a combinatorial experiment to an end 
of a combinatorial experiment. The regions of interest are 
identified after the series of images has been captured (160). 
The images are infrared images (100). Regions of interest 
are identified by a user and have a user-defined shape (170, 
190). A graphical representation of the figures of merit is 
displayed (196). 
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ANALYSIS OF CHEMICAL DATA FROM IMAGES 

A portion of the disclosure of this patent document contains 
5 material which is subject to copyright protection. The copyright 

owner has no objection to the ■ facsimile reproduction by anyone of the 
patent document or the patent disclosure, as it appears in the Patent 
and Trademark Office patent file or records, but otherwise reserves 
all copyright rights whatsoever. 

10 ' CROSS REFERENCE TO RELATED APPLICATIONS 

This application is a continuation-in-part of commonly 
assigned, co-pending U.S. Patent Application Serial No. 09/227,558, 
filed January 8, 1999, which is a continuation-in-part of U.S. 
Application Serial No. 08/898,715, filed July 22, 1997, and U.S. 

15 Provisional Application Nos . 60/050,949, filed June 13, 1997; 

60/028,106, filed October 9, 1996; 60/029,255, filed October 25, 
1996; 60/035,366, filed January 10, 1997; 60/048,987, filed June 9, 
1997; 60/028,105, filed October 9, 1996; and 60/035,202, filed 
January 10, 1997. Each of the foregoing applications is incorporated 

20 herein by reference in its entirety. 

BACKGROUND OF THE INVENTION 
The present invention relates to methods and computer 

programs for rapidly screening and characterizing materials by the 

analysis of data acquired from images. 
25 In combinatorial chemistry, a large number of candidate 

materials are created from a relatively small set of precursors and 

subsequently evaluated for suitability for a particular application. 

As currently practiced, combinatorial chemistry permits scientists to 

systematically explore the influence of structural variations in 
30 candidates by dramatically accelerating the rates at which they are 

created and evaluated. Compared to traditional discovery methods, 

combinatorial methods sharply reduce the costs associated with 

preparing and screening each candidate. 

Combinatorial chemistry has revolutionized the process of 
35 drug discovery. One can view drug discovery as a two-step process: 

acquiring candidate compounds through laboratory synthesis or through 
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natural products collection, followed by evaluation or screening for 
efficacy. Pharmaceutical researchers have long used high-throughput 
screening (HTS) protocols to rapidly evaluate the therapeutic value 
of natural products and libraries of compounds synthesized and 
5 cataloged over many years. However, compared to HTS protocols, 

chemical synthesis has historically been a slow, arduous process. 
With the advent of combinatorial methods, scientists can now create 
large libraries of organic molecules at a pace on par with HTS 
protocols. 

10 Recently, combinatorial approaches have been used for 

discovery programs unrelated to drugs. For example, some researchers 
have recognized that combinatorial strategies also offer promise for 
the discovery of inorganic compounds such as high-temperature 
superconductors, magneto resistive materials, luminescent materials, 

15 and catalytic materials. See, for example, U.S. Patent No. 5,776,359 
and co-pending U.S. patent application no. 08/898,715 "Combinatorial 
Synthesis and Analysis of Organometallic Compounds and Catalysts" 
(published as WO 98/03251), which are both incorporated herein by 
reference . 

20 SUMMARY OF THE INVENTION 

The invention provides computer programs and computer- 
implemented methods for extracting and analyzing combinatorial 
chemical data from images. 

In general, in one aspect, the invention features a computer 

25 program for evaluating a combinatorial library including a plurality 
of members. The program includes instructions to receive a stream of 
data including a series of images of the combinatorial library; to 
identify a plurality of regions of interest, each region 
corresponding to a location in each of the series of images and to a 

30 location in the combinatorial library; to determine a series of 

values for one or more regions of interest, the series of values for 
each of the one or more regions including a value for each of the 
images; and to calculate from each series of values for the one or 
more regions a figure of merit for the library member at the 
35 corresponding library location. 

Implementations of the invention can include one or more of 
the following advantageous features. Each region of interest 
corresponds to a plurality of pixels in each image and the series of 
values for a region of interest comprises an average value for each 
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of the corresponding plurality of pixels in each of the images. The 
stream of data comprises a series of images generated at a frequency 
of greater than about 1 frame per second. The stream of data 
comprises a series of images generated at a frequency of greater than 
5 about 6 frames per second. The stream of data comprises a series of 
images generated at a frequency of greater than about 12 frames per 
second. The stream of data comprises a series of images generated at 
a frequency of greater than about 20 frames per second. The program 
further includes instructions to display a graphical representation 
10 of the figures of merit. The graphical representation includes a 
histogram. The series of images is captured from a beginning of a 
combinatorial experiment to an end of the combinatorial experiment 
and the plurality of regions of interest may be identified after the 
series of images has been captured or after one or more images in the 
15 series of images has been captured. The series of images includes a 
series of infrared images. The figure of merit comprises an 
emittance change for a library member. The figure of merit comprises 
a phase transition point for a library member. The figure of merit 
comprises a thermoelectric figure of merit for a library member. The 
20 instructions to identify a plurality of regions of interest comprise 
instructions to receive a first user input identifying one or more 
regions of interest, each region having a user-defined shape. The 
program further includes instructions to receive a second user input 
defining a computation for determining a series of values for each of 
25 the one or more regions of interest. The instructions to determine a 
series of values for each of the one or more regions of interest 
include instructions to compute each value in each series in 
accordance with the second user input. The computation is an average 
for a group of pixels corresponding to a region of interest. 
30 In general, in another aspect, the invention features a 

computer program on a computer-readable medium for evaluating a 
combinatorial library including a plurality of members. The program 
includes instructions to receive a stream of data including a series 
of images of the combinatorial library; to receive a first user input 
35 identifying one or more regions of interest, each region having a 
user-defined shape and each region corresponding to a location in 
each of the series of images and to a location in the combinatorial 
library; to receive a second user input defining a computation for 
determining a series of values for each of the one or more regions of 
40 interest, each series of values including a value for each of the 
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images; and to determine a series of values for each of the one or 
more regions of interest, each value in each series being computed in 
accordance with the second user input. 

Implementations of the invention can include one or more of 
the following advantageous features. The program further includes 
instructions to calculate from each series of values for the one or 
more regions a figure of merit for the library member at the 
corresponding library location. 

In general, in another aspect, the invention features a 
computer program for evaluating a combinatorial chemical experiment. 
The program includes instructions to receive a series of images of a 
combinatorial library, the library including a plurality of members, 
the series of images captured from a beginning of the experiment to 
an end of the experiment; to identify, after the series of images has 
been captured, a plurality of regions of interest, each region 
corresponding to a location in each of the series of images and to a 
location in the combinatorial library; and to determine a series of 
values for one or more regions of interest, the series of values for 
each of the one or more regions including a value for each of the 
images. 

In general, in another aspect, the invention features a 
method for evaluating a combinatorial library including a plurality 
of members. The method includes receiving a stream of data including 
a series of images of the combinatorial library; identifying a 
plurality of regions of interest, each region corresponding to a 
location in each of the series of images and to a location in the 
combinatorial library; determining a series of values for one or more 
regions of interest, the series of values for each of the one or more 
regions including a value for each of the images; and calculating 
from each series of values for the one or more regions a figure of 
merit for the library member at the corresponding library location. 

In general, in another aspect, the invention features a 
computer-implemented method for evaluating a combinatorial chemical 
experiment. The method includes capturing a series of images of a 
combinatorial library, the library including a plurality of members, 
the series of images being captured from a beginning of the 
experiment to an end of the experiment and storing the series of 
images in memory; after the series of images has been captured, 
identifying a plurality of regions of interest, each region 
corresponding to a location in each of the series of images and to a 
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location in the combinatorial library; and determining a series of 
values, for one or more regions of interest, the series of values for 
each of the one or more regions including a value for each of the 
images . 

5 In general, in other aspect, the invention features a system 

for evaluating a combinatorial library including a plurality of 
members. The system includes means for receiving a stream of data 
comprising a series of images of the combinatorial library; means for 
identifying a plurality of regions of interest, each region 

10 corresponding to a location in each of the series of images and to a 
location in the combinatorial library; means for determining a series 
of values for one or more regions of interest, the series of values 
for each of the one or more regions comprising a value for each of 
the images; and means for calculating from each series of values for 

15 the one or more regions a figure of merit for the library member at 
the corresponding library location. 

In general, in other aspect, the invention features a system 
for evaluating a combinatorial library, the library including a 
plurality of members. The system includes means for receiving a 

20 stream of data comprising a series of images of ..the combinatorial 
library; means for receiving a first user input identifying one or 
more regions of interest, each region having a user-defined shape and 
each region corresponding to a location in each of the series of 
images and to a location in the combinatorial library; means for 

25 receiving a second user input defining a computation for determining 
a series of values for each of the one or more regions of interest, 
each series of values comprising a value for each of the images; and 
means for determining a series of values for each of the one or more 
regions of interest, each value in each series being computed in 

30 accordance with the second user input. 

In general, in other aspect, the invention features a system 
for evaluating a combinatorial chemical experiment. The system 
includes means for receiving a series of images of a combinatorial 
library, the library including a plurality of members, the series of 

35 images captured from a beginning of the experiment to an end of the 
experiment; means for identifying a plurality of regions of interest 
after the series of images has been captured, each region 
corresponding to a location in each of the series of images and to a 
location in the combinatorial library; and means for determining a 

40 series of values for one or more regions of interest, the series of 
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values for each of the one or more regions comprising a value for 
each of the images. 

Advantages that can be seen in implementations of the 
invention include one or more of the following. The use of a video 

5 image sequence allows the automated extraction of data from one or 
more regions in every frame of the image set. This results in a 
time-resolved profile of the reaction or transformation being 
observed, rather than merely snapshots based on a limited number of 
selected images or image frames. Regions of interest in each image 

10 can be defined after the image sequence is captured. Calculations 

and corrections can be applied automatically to every region in every 
frame of the image set. One or more figures of merit can be 
extracted from some or all of the data set for each region and can be 
graphically displayed. 

15 The details of one or more embodiments of the invention are 

set forth in the 

accompanying drawings and the description below. Other features and 
advantages of the invention will become apparent from the 
description, the drawings, and the claims. 

20 BRIEF DESCRIPTION OF THE DRAWINGS 

The file of this patent contains at least one drawing 
executed in color. Copies of this patent with color drawing (s) will 
be provided by the Patent and Trademark Office upon request and 
payment of the necessary fee. 
25 FIG. 1 is a block diagram of a data processing system 

showing an implementation of the invention. 

FIG. 2 is a flow diagram of a method of reducing a data 

stream. 

FIG. 3 is a flow diagram illustrating the method of FIG. 2 

30 in more detail. 

FIG. 4 is a flow diagram of a method of generating a mask. 

FIG. 5A. is a diagram of a combinatorial library. 

FIG. 5B is an illustration of a dialog window for creating a 

mask. 

35 FIG. 5C is an illustration of selecting the corners of a 

mask. 

FIG. 6 is a flow diagram of a method of analyzing reduced 
data values and displaying calculated results. 
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FIG. 7 is a flow diagram of a method of selecting display 
parameters and a display format. 

FIGS. 8A-8C are illustrations of formats for displaying 
experimental results. 

FIG. 9 is a window displaying images in data stream. 
FIG . 10 is a schematic diagram of a thermoelectric device. 
FIG. 11 is a schematic diagram of a thermoelectric device 
including a combinatorial array of thermoelectric materials. 

FIGS. 12A-12B are illustrations of display formats for a 
combinatorial thermoelectric materials experiment. 

FIG. 13 is a schematic diagram of a combinatorial experiment 
to monitor heats of reaction. 

FIG. 14 is a diagram illustrating the combinatorial library 
of FIG. 13 in more detail. 

FIG. 15 is a diagram of a combinatorial experiment to 
determine phase transition points of a library of materials. 

FIG. 16 is a schematic diagram of a computer platform 
suitable for implementing the data processing system of the 
invention . 

Like reference numbers and designations in the various 
drawings indicate like elements. 

DETAILED DESCRIPTION 
In the present invention, a camera observes an experiment. 
The resulting image data is analyzed by a data processing system 
25 implementing methods of the present invention, as will be described. 

Referring to FIG. 1, camera 100 observes experiment 110 and 
records one or more images representing experiment 110. In the. 
exemplary system that will be described, camera 100 is a commercially 
available high performance infrared camera, such as the SE-IR 
30 CamlRa™, available from SE-IR Corporation of Goleta, California. 
Such cameras can be used to measure thermal properties such as 
radiant or photon emittance, exitance, thermal flux and the like. 
Commercially available software, such as SE-IR Corporation's CamIRa™ 
software package, can be used to drive the camera's operations and to 
35 generate the images. Experiment 110 can be any experiment suitable 
for monitoring with camera 100. It can involve a single reaction 
vessel containing reagents or a single material deposited on a 
substrate within the field of view of camera 100. Alternatively, it 
can include a combinatorial array or "library" of distinct "members" 
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(separate vessels or deposited materials), as will be discussed in 
more detail below. 

Image data 120 generated by camera 100 is stored in data 
store 130 as an array of picture elements or "pixels," each of which 
is represented by a specific intensity or pixel value. Data 
processing system 140 retrieves image data 120 from data store 130; 
alternatively, data processing system 140 can receive image data 120 
directly from camera 100 without intervening storage in data store 
130. After retrieving the image data, data processing system 140 
processes the image data in data reduction module 150 and data 
analysis module 160 either automatically or under user control. A 
user can interact with system 140 through user interface module 170. 
Data processing system 14 0 displays results through display or 
printing devices 190, 196. 

As shown in FIG. 2, data reduction module 150 begins by 
getting a stream of data comprising a series of images from data 
store 130 or camera 100 (step 200) . After getting the first image in 
the stream (step 210), data reduction module 150 reduces the data 
from that image, as will be described in further detail below (step 
220). If additional images remain to be reduced (step 230), data 
reduction module 150 gets the next image (step 240) and reduces that 
image data (step 220) . Data reduction module 150 provides the 
reduced data values to data analysis module 160 (step 250). 
Optionally, data processing system 140 stores the reduced data in 
data store 130. 

FIG. 3 describes data reduction step 220 in more detail. 
After receiving an image (step 210), data reduction module 150 
obtains a user-defined mask identifying one or more regions of 
interest in the image (step 300) . These regions may correspond, for 
example, to reaction vessels or materials that make up the elements 
of a combinatorial library used in experiment 110. Data reduction 
module 150 applies the mask to the image (step 310), and extracts a 
value or values for each region of interest (step 320) . Data 
reduction module 150 uses these values to calculate a reduced data 
value for each region of interest in the image (step 330) . This 
reduced data value can be an average of intensity values recorded by 
camera 100 for points within a region of interest. It can also be an 
average change in intensity calculated by subtracting a reference 
point value, for example, a value measured for the region at an 
earlier time (e.g., in a previous image), a value measured for 
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background noise or some other value (such as reflectance or base 
line) . In other embodiments, the reduced data value can be 
calculated using other known statistical functions. Optionally, data 
reduction module 150 allows the user to select a desired function, 

5 for example by choosing from a menu of possible functions. Data 

reduction module 150 produces a list of reduced data values for the 
array of regions of interest, which is sent to data analysis module 
160 (step 340) . The list of reduced data values may also be stored 
for future use. Data reduction module 150 then determines whether 

10 additional images remain to be reduced (step 230) . 

The process of creating a mask is described in FIG. 4. 
Through user interface 170, the user identifies one or more regions 
of interest within an image (step 400). As discussed above, these 
regions will generally correspond to specific reaction vessels or 

15 materials that make up the elements of a combinatorial library 

monitored by camera 100. Optionally, the user can also specify one 
or more reference regions to which experimental data can be compared 
(step 410) . Data processing system 140 saves the resulting mask in 
data store 130 for subsequent use during experiment 110, as well as 

20 for use during later experiments sharing a similar substrate geometry 
(step 420) . 

FIGS. 5A-5C detail the creation of a mask in one embodiment 
for an experiment involving a combinatorial library 500 of members 
510 (FIG. 5A) . User interface 170 displays "Create Mask" dialog 

25 window 520 (FIG. 5B) . The user can select a grid type 530, as well 

as the number of horizontal and vertical members in the grid, 540 and 
550, respectively. The user can also select a member type 560, 
according to the nature of experiment 110. In another embodiment, 
data reducing module 150 allows the user to define the shape of mask 

30 regions to correspond to any shape of library members 510, for 
example, through a user interface similar to those found in 
commercially available drawing and painting software known to those 
skilled in the art. 

While FIG. 5B shows a "Create Mask" dialog for 

35 thermoelectric materials, the nature of the experiment is not a 

limiting feature for this invention as discussed below. The user can 
select a particular member length or width 570. After selecting the 
"Create" button, the user graphically specifies three corners 580 of 
the grid, clicks on the image and connecting lines are drawn (FIG. 

40 5C) . The elements of the grid can be moved as groups or individually 
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to accommodate spacial irregularities in the physical substrate. 
Data reduction module 150 uses this information to create the mask. 

As shown in FIG. 6, data analysis module 160 gets the list 
of reduced data values for each image from data reduction module 150 
or data store 130 (step 600). Data analysis module 160 uses the 
values to calculate experimental results, including one or more 
figures of merit for each region of interest (step 610) , as will be 
described in more detail below. The user can save the calculated 
results (steps 620 and 630), and can view the displayed results in a 
number of graphical formats (steps 620 and 640) . 

The user's interaction with data analysis module 160 (steps 
620 through 640) is described in more detail in reference to FIG. 7. 
Through user interface 170, the user can select a variety of display 
options (step 700) . These include a number of display formats (step 
710) illustrated in FIGS. 8A-8C, such as an array of "thumbnail" 
graphs 800 plotting experimental data (for example, intensity versus 
time) for each region of interest a graph of experimental data for 
any selected region of interest (a "zoom-in" graph) 850, or a three- 
dimensional histogram 880 displaying a calculated value for each 
region of interest. Data analysis module retrieves display 
parameters and control information (step 720) and gets the results 
calculated at step 610 (step 730). Data analysis module 160 displays 
the results in the selected display format (step 740). User 
selection step 700 also includes the option to choose display 
parameters (step 750) . These include parameters such as histogram 
scale, range and colors, or time and value ranges for thumbnail or 
zoom-in graphs. By setting these parameters, the user can view 
experimental results over a specified time period, with the value at 
any specified time subtracted to show changes in temperature, or with 
a reference value (as described above) subtracted to remove 
background (or systematic) changes. 

As shown in FIG. 9, the user can also view the original 
image or images recorded by camera 100. Window 900 displays an 
infrared image 910 of experiment 110, such as microtiter plate 920. 
Depending on the user-selected display parameters, different colors 
in infrared image 910 depict the range of heat emitted by objects 
within the field of camera 100. The user can navigate through the 
stream of images using buttons 930a-e provided in pane 940. Buttons 
930d and 930b navigate forward or backward, respectively, by a single 
frame. Buttons 930e and 930a provide "fast forward" and "rewind" 
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functions, respectively. In one embodiment, in "fast forward" mode 
data analysis module 160 and/or user interface module 170 determines 
the number of frames remaining in the stream of images and displays 
selected frames until it reaches the end of the stream. Similarly, 
in "rewind" mode, the number of preceding frames is determined and 
selected frames are displayed in reverse order until the beginning of 
the image stream is reached. Thus, for example, fast forward mode 
can display every tenth frame in a stream including 1000 frames, for 
a total of 100 frames displayed. Pane 940 also displays the frame 
number of the image displayed. 

Data processing system 140 can be used to analyze material 
or chemical systems that lend themselves to characterization by IR 
thermography. Using IR thermography, the emittance (or exitance) of 
an entire library of materials can be monitored with an infrared 
camera and the measured values can be used to derive a variety of 
thermodynamic properties associated with the materials. Commercial 
position sensitive systems such as infrared focal plane arrays can 
have a high sensitivity over a wide temperature range. Commercial 
infrared cameras incorporating such systems can acquire data at 
speeds up to 100 or more frames per second. In combination with such 
a system, data processing system 140 is particularly useful as a tool 
for quickly screening large numbers of materials for desirable 
thermodynamic characteristics. In one embodiment, data processing 
system 140 receives images at a rate of 12 to 20 frames per second 
and reduces the data by averaging values for each frame to obtain the 
equivalent of one averaged frame of data per second, thereby 
increasing the signal to noise ratio of the experimental results. 

Depending on the process being monitored, different data 
acquisition rates — for example, rates of about 1, 6, 12 or 20 
frames per second — provide a data stream suitable for deriving a 
reaction profile sufficient to track many chemical reactions, 
material transformations and thermal diffusion transients. Higher 
data acquisition rates are useful in characterizing processes 
requiring a relatively high degree of resolution. For example, metal 
compositions can be characterized based on the range of temperatures 
at which melting occurs. For many metal compositions, melting begins 
at one temperature and continues until the material reaches a higher 
temperature. The first temperature, below which the entire sample is 
in a solid phase, is known as the solidus temperature; the second, 
above which the entire sample is in a liquid phase, is known as the 
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liquidus temperature. To identify eutectic compositions — those in 
which the solidus temperature and liquidus temperature are equal — 
it is useful to screen an array of metals covering a range of 
compositions for those having the lowest liquidus temperature. By 
rapidly heating such an array, it is possible to exceed a 
composition's solidus temperature without the onset of melting due to 
slow diffusion in the solid state. If heating is sufficiently rapid 
— for example at 20° to 60°C per second — it is possible to reach 
the liquidus temperature without melting, at which point the entire 
sample quickly melts. Data acquisition at speeds of 18 to 20 frames 
per second are sufficient to resolve such rapid thermodynamic 
changes . 

The features and advantages of the invention can be 
appreciated from a description of its application to a particular 
implementation in the field of infrared thermography — the 
characterization of thermoelectric materials. 

Thermoelectric Materials 

Thermoelectric devices are solid state devices that pump 
heat from one junction to another when subjected to an electric 
current, a phenomenon known as the Peltier effect. Thermoelectric 
materials are characterized by a number of physical parameters 
including the thermal conductivity k, the electrical conductivity a, 
the Seebeck coefficient S, the Hall coefficient R H , the charge 
carrier mobility u, the device operating temperature T, the charge 
carrier effective mass m*, and the band gap E fl . An estimate of the 
thermoelectric device efficiency for pumping heat relative to the 
heat lost due to the electrical resistivity is the thermoelectric 
figure of merit defined by: 

K 

FIG. 10 illustrates a model thermoelectric device 
constructed from a thermoelectric material 1000 connected to voltage 
source 1010 with voltage wires 1020 made of a high conductivity 
metal. Under steady state or adiabatic conditions, the heat pumped 
by the Peltier effect will be equal to the heat carried by the 
thermal conduction, which leads to the following relationship for the 
thermoelectric figure of merit: 
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Qpeltier _ ^Peltier 
Joule Joule 



Qpeitier and Q Joule are the amount of heat transported by the Peltier 
effect and the amount of energy lost to Joule heating, respectively; 
Ppeitier and P Jeule are each the corresponding 
5 power (heat per unit time) . The expression for the total power 

dissipated in the device is given by the sum of the thermoelectric 
(Peltier) and Joule components: 

V 2 V 

p - p + p = _ + T7-- 

r r Joule r Peltier ^ " g 

where the first term is the power dissipated by Joule effects (P j0 m e ) 

10 and the second term is the contribution from the Peltier component 
(P Peltl „); R is the electrical resistance of the circuit and n is the 
Peltier coefficient of the thermoelectric material measured relative 
to the material that makes up the voltage contacts. Application of 
an oscillatory voltage at a reference frequency o 0 , such as V(t) = 

15 V 0 cos(G3 0 t), causes the Joule term to oscillate at twice the reference 
frequency (2co 0 ) due to the voltage being squared, while the Peltier 
term is linear in voltage and is observed at the reference frequency 
co 0 . Measurement of the power dissipated at the junction between the 
voltage contact and the thermoelectric material as a function of 

20 time, P(t), followed by a Fourier transform to power as a function of 
frequency, P(co), allows the contribution due to Joule effects to be 
distinguished from (and compared to) the contribution due to Peltier 
effects. The ratio of the two amplitudes P (w 0 ) /P (2co 0 ) is the 
thermoelectric figure of merit ZT. 

25 The present invention provides a fast and efficient screen 

for the identification of thermoelectric materials in a combinatorial 
library of candidate materials, as well as* an analytical tool for 
characterizing such materials by determining the thermoelectric 
figure of merit. Suitable libraries can be created using techniques 

30 such as those described in U.S. Patent No. 5,77 6,359, which is 
incorporated herein by reference in its entirety. 

As shown in FIG. 11, in combinatorial thermoelectric device 
1100, voltage source 1110 applies a parallel voltage through voltage 
wires 1120 to all members 1130 in a combinatorial library 1140. 

35 Library 1140 is made from a material with low thermal conductance, 
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such as 25 to 50 micron thick polyimide sheets to minimize heat lost 
to the substrate holding elements 1130; for the same reason, library 
1140 is maintained in an evacuated environment to minimize heat lost 
to the surroundings. To maintain adiabatic conditions, the voltage 
5 applied by voltage source 1110 is kept as small as possible, with 

reference frequencies typically being on the order of 0.1 Hz or less. 
Voltage wires 1120 are made of a high conductivity metal such as Ag, 
Au, Cr, Ta or Cu to maintain good electrical contact and because 
their Peltier coefficients are low relative to those found in 
10 semiconducting materials. 

As discussed more generally above, data reduction module 150 
obtains a series of images of thermoelectric device 1100 from 
infrared camera 100. Data reduction module 150 generates reduced 
data values as described above, and provides those values to data 
15 analysis module 160. 

With these reduced data values, data analysis module 160 
calculates the intensity as a function of frequency for each region 
of interest using fast Fourier transform. As shown in FIGS. 12A and 
12B, the user can view the data graphically, in either the time 
20 domain 1200 or the frequency domain 1250. .Data analysis module. 160 
uses the frequency data to calculate the thermoelectric figure of 
merit for each region of interest as described above. The figure of 
merit (1220, 1270) is displayed in thumbnail graphs 1210 and 1260. 
The user can also view a three dimensional histogram displaying the 
25 figure of merit for each region of interest, as described above. 

Chemical Transformations 
Data processing system 140 can also be used to monitor the 
progress of chemical reactions or transformations. In many 
embodiments, the reaction is a catalytic reaction and catalyst 
30 activity is being determined, which is the example described below. 
Referring to FIG. 13, in this application an infrared camera 100 
monitors emittance associated with heat evolution or absorption or 
with a compound or chemical's disappearance from or appearance in a 
combinatorial library 1310 under various external conditions such as 
35 temperature and gas flow. For example, if a solid catalyst library 
and its surrounding support in a two-dimensional library are exposed 
to a reactant, a measurable heating of the surroundings may occur 
depending on the activity of the chemical process. Such libraries 
can be generated using techniques such as those disclosed in U.S. 
40 Applications Serial No. 08/898,715, filed July 22, 1997, or Serial 
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No. 09/227,558, filed January 8, 1999, which are both incorporated 
herein by reference in their entirety. In the case of a catalyst, 
the activity of the catalyst correlates to the energy released or 
absorbed as heat during the chemical reaction between the catalyst 
and the exchange gas. In a combinatorial library 1310, members 1320 
are nearly identical in thermal mass, so measurements of the heat 
evolved by one element in the library relative to others within the 
library can be used to characterize the chemical processes induced by 
these materials. 

Figure 14 illustrates a two-dimensional library 1400 of 
materials useful in this embodiment. The individual library members 
are reaction wells 1410 in a substrate 1420. Substrate 1420 is 
placed within a sealed reaction chamber which is pressurized with the 
relevant gas. Windows 1430 and 1440 are made of an infrared 
transparent medium (such as sapphire, antiref lection coated silicon, 
BaF 2 , CaF 2 or NaCl) capable of maintaining the pressurized gas inside 
the chamber. Since windows 1430 and 1440 are transparent to infrared 
radiation, thermal imaging techniques can be used to monitor the heat 
of reaction of each library element under various external 
conditions. 

Measuring the heat generated by a catalytic transformation 
is a powerful technique for rapidly screening catalyst efficacy. For 
condensed phase reactions, there is direct thermal contact between 
the catalyst, products, and solvent. Thus, the library can be 
directly imaged and the emissivity of the samples will be roughly 
that of the pure solvent, allowing direct comparison of the members 
in a library. In cases where emissivity differences exist between 
materials in the library, direct comparison of library members is 
more complicated, but the experimental results are still useful in 
screening for raw catalyst activity. Differences in emissivity may 
also be dealt with by imaging the library through a material (such as 
graphite) . This technique is particularly useful for screening gas 
phase reactions, where the catalyst is mounted on a support and large 
emissivity differences are common. In this case, the reaction can be 
monitored from the backside of the catalyst support. 

In a typical experiment, a library is loaded into a sample 
chamber with an IR transparent window and the entire system is 
allowed to come to thermal equilibrium. The sample is monitored with 
camera 100 to establish a background reading, and the chamber is 
pressurized with a reactant gas. The sample is monitored by camera 
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100 over the course of the catalytic reaction, for example, for about 
one hour. The resultant data is analyzed as will be described below. 
The change in emittance is directly proportional to the catalytic 
activity of the library member. 

As discussed more generally above, data reduction module 150 
obtains a series of images of library 1400 from infrared camera 100. 
Data reduction module 150 generates reduced data values as described 
above, and furnishes those values to data analysis module 160. 

With these reduced data values, data analysis module 160 
calculates the integral of intensity (proportional to the total 
conversion of starting material) . As described above, the user can 
view the data in the form of an array of graphs 800 of intensity 
versus time and integral of intensity versus time. Each thumbnail 
graph also depicts two figures of merit for each region of interest - 
- the total integrated intensity and the maximum intensity for the 
element. The user can also view a three dimensional histogram 
displaying either figure of merit for each region of interest. 

Metals 

Data processing system can also be used to characterize 
melting points of metals or other materials (e.g., composite 
materials) using infrared imaging. In this application, an infrared 
camera 100 is used to monitor a library whose members include a 
series of metals or other materials. Suitable libraries can be 
generated using techniques such as those described in U.S. Patent 
No. 5,776,359, which is incorporated herein by reference in its 
entirety. Camera 100 measures intensities that are a function of 
temperature and emissivity of each member. Changes in the structure 
and bonding of a chemical composition during a transition from one 
thermodynamic phase to another result in a change in the 
composition's emissivity. During a phase transition the temperature 
of a library member may change or the rate of temperature change may 
increase or decrease. The thermodynamics of a given material's phase 
transition can be characterized by comparing the emissivity changes 
over a given temperature range with that of a standard material 
having a constant emissivity in the relevant temperature range. 

The infrared camera 100 monitors the intensity from every 
library member in parallel and compares it to the intensity of a 
known standard material within the field of view of the camera (which 
may be the substrate holding the library) and subjected to the same 
physical conditions as the library members. In this way, complicated 
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phase relationships are measured for large libraries of materials by 
heating or cooling the library and deriving changes in the 
emissivity . 

Referring to FIG . 15, as discussed more generally above, in 

5 this embodiment camera 100 records infrared images of library 1500, 
including library members 1510, as the temperature is varied over a 
range of interest. Data reduction module 150 obtains a series of 
images of library 1500 and generates reduced data values as described 
above. Data reduction module 150 furnishes those values to data 

10 analysis module 160. 

Using these reduced data values, data analysis module 160 
calculates the relative emissivity of each library member as compared 
to the standard. After smoothing the data, data analysis module 160 
calculates the derivative of the ratio of library member emissivity 

15 to standard. The maximum rate of change occurs at the material's 

melting point which here is the figure of merit. As described above, 
the user can view the experimental results as an array of thumbnail 
graphs 800 depicting a plot of intensity versus time for a given 
library member, intensity versus time for the standard sample (which 

20 may be the substrate) and a plot. that is a ratio of the first two 

plots. The user can also view a reduced plot of melting point as a 
function of library member. The figure of merit ~ the melting point 
calculated as described above — is also displayed. As described 
above, the user can also view the data as a three dimensional 

25 histogram displaying either figure of merit for each library element. 

The invention can be implemented in digital electronic 
circuitry, or in computer hardware, firmware, software, or in 
combinations of them. Apparatus of the invention can be implemented 
in a computer program product tangibly embodied in a machine-readable 

30 storage device for execution by a programmable processor; and method 
steps of the invention can be performed by a programmable processor 
executing a program of instructions to perform functions of the 
invention by operating on input data and generating output. The 
invention can be implemented advantageously in one or more computer 

35 programs that are executable on a programmable system including at 
least one programmable processor coupled to receive data and 
instructions from, and to transmit data and instructions to, a data 
storage system, at least one input device, and at least one output 
device. Each computer program can be implemented in a high-level 

40 procedural or object-oriented programming language, or in assembly or 
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machine language if desired; and in any case, the language can be a 
compiled or interpreted language. Suitable processors include, by 
way of example., both general and special purpose microprocessors. 
Generally, a processor will receive instructions and data from a 

5 read-only memory and/or a random access memory. Storage devices 
suitable for tangibly embodying computer program instructions and 
data include all forms of non-volatile memory, including by way of 
example semiconductor memory devices, such as EPROM, EE PROM, and 
flash memory devices; magnetic disks such as internal hard disks and 

10 removable disks; magneto-optical disks; and CD-ROM disks. Data can 
also be temporarily stored in volatile memory. Any of the foregoing 
can be supplemented by, or incorporated in, ASICs 
(application-specific integrated circuits) . 

To provide for interaction with a user, the invention can be 

15 implemented on a computer system having a display device such as a 
monitor or LCD screen for displaying information to the user and a 
keyboard and a pointing device such as a mouse or a trackball by 
which the user can provide input to the computer system. The 
computer system can be programmed to provide a graphical user 

20 interface through which computer programs interact with users. 

An example of one such type of computer is shown in FIG. 16, 
which shows a block diagram of a programmable processing system 
(system) 1600 suitable for implementing or performing the apparatus 
or methods of the invention. The system 1600 includes a processor 

25 1610, a random access memory (RAM) 1620, a program memory 1630 (for 
example, a writable read-only memory (ROM) such as a flash ROM) , a 
hard drive controller 1640, and an input /output (I/O) controller 1650 
coupled by a processor (CPU) bus 1660. The system 1600 can be 
preprogrammed, in ROM, for example, or it can be programmed (and 

30 reprogrammed) by loading a program from another source (for example, 
from a floppy disk, a CD-ROM, or another computer) . 

The hard drive controller 1640 is coupled to a hard disk 
1645 suitable for storing executable computer programs, including 
programs embodying the present invention, and data including the 

35 images, masks, reduced data values and calculated results used in and 
generated by the invention. 

The I/O controller 1650 is coupled by means of an I/O bus 
1670 to an I/O interface 1680. The I/O interface 1680 receives and 
transmits data (e.g., stills, pictures, movies, and animations for 

40 importing into a composition) in analog or digital form over 
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Also coupled to the I/O bus 1670 is a display 1690 and a keyboard 1695. 
Alternatively, separate connections (separate buses) can be used for the I/O interface 1670, 
display 1690 and keyboard 1695. 

The invention has been described in terms of particular embodiments. Other 
embodiments are within the scope of the following claims. For example, the invention has 
been described as being implemented in a number of different embodiments, each intended to 
monitor, analyze and display data for a different experiment. Alternatively, a number of 
different applications can be implemented in a single system, for example, where data 
processing system 140 includes multiple versions of data analysis module 140, each designed 
to analyze and display data from a different type of experiment. Also, data structures other 
than the ones mentioned above can be used in storing and processing data. For example, 
mask and image information can be encapsulated in objects and stored in an object oriented 
database. In addition, the steps of the invention can be performed in a different order and 
still achieve desirable results. 
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What is claimed is: 

1 . A computer program on a computer-readable medium for evaluating a combinatorial 
library, the library including a plurality of members, the program comprising instructions to: 

receive a stream of data comprising a series of images of the combinatorial library; 

identify a plurality of regions of interest, each region corresponding to a location in 
each of the series of images and to a location in the combinatorial library; 

determine a series of values for one or more regions of interest, the series of values 
for each of the one or more regions comprising a value for each pf the images; and 

calculate from each series of values for the one or more regions a figure of merit for 
the library member at the corresponding library location. 

2. The computer program of claim 1 , wherein each region of interest corresponds to a 
plurality of pixels in each image and the series of values for a region of interest comprises an 
average value for each of the corresponding plurality of pixels in each of the images. 

3. The computer program of claim 2, wherein the stream of data comprises a series of 
images generated at a frequency of greater than about 1 frame per second. 

4. The computer program of claim 2, wherein the stream of data comprises a series of 
images generated at a frequency of greater than about 6 frames per second. 

5. The computer program of claim 2, wherein the stream of data comprises a series of 
images generated at a frequency of greater than about 12 frames per second. 

6. The computer program of claim 2, wherein the stream of data comprises a series of 
images generated at a frequency of greater than about 20 frames per second. 

7. The computer program of claim 2, further comprising instructions to display a 
graphical representation of the figures of merit. 
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1 8. The computer program of claim 7, wherein the graphical 

2 representation comprises a histogram. 

1 9. The computer program of claim 1, wherein the series of images 

2 is captured from a beginning of a combinatorial experiment to an end 

3 of a combinatorial experiment and wherein the plurality of regions of 

4 interest is identified after the series of images has been captured. 

1 10. The computer program of claim 1, wherein the series of images 

2 comprises a series of infrared images. 

1 11. The computer program of claim 10, wherein the figure of merit 

2 comprises an emittance change for a library member. 

1 12. The computer program of claim 10, wherein the figure of merit 

2 comprises a phase transition point for a library member. 

1 13. The computer program of claim 10, wherein the figure of merit 

2 comprises a thermoelectric figure of merit for a library member. 

1 14. The computer program of claim 1, wherein the instructions to 

2 identify a plurality of regions of interest comprise instructions to 

3 receive a first user input identifying one or more regions of 

4 interest, each region having a user-defined shape, the computer 

5 program further comprising instructions to receive a second user 

6 input defining a computation for determining a series of values for 

7 each of the one or more regions of interest. 

1 15. The computer program of claim 14, wherein the instructions to 

2 determine a series of values for each of the one or more regions of 

3 interest comprise instructions to compute each value in each series 

4 in accordance with the second user input. 

1 16. The computer program of claim 15, wherein the computation is 

2 an average for a group of pixels corresponding to a region of 

3 interest. 
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1 17. A computer program on a computer-readable medium for 

2 evaluating a combinatorial library, the library including a plurality 

3 of members, the program comprising instructions to: 

4 receive a stream of data comprising a series of images of the 

5 combinatorial library; 

6 receive a first user input identifying one or more regions of 

7 interest, each region having a user-defined shape and each region 

8 corresponding to a location in each of the series of images and to a 

9 location in the combinatorial library; 

10 receive a second user input defining a computation for 

11 determining a series of values for each of the one or more regions of 

12 interest, each series of values comprising a value for each of the 

13 images; and 

14 determine a series of values for each of the one or more 

15 regions of interest, each value in each series being computed in 

16 accordance with the second user input. 

1 18. The computer program of claim 17, further comprising 

2 instructions to calculate from each series of values for the one or 

3 more regions a figure of merit for the library member at the 

4 corresponding library location. 

1 19. A computer program for evaluating a combinatorial chemical 

2 experiment, the program comprising instructions to: 

3 receive a series of images of a combinatorial library, the 

4 library including a plurality of members, the series of images 

5 captured from a beginning of the experiment to an end of the 

6 experiment; 

7 after the series of images has been captured, identify a 

8 plurality of regions of interest, each region corresponding to a 

9 location in each of the series of images and to a location in the 

10 combinatorial library; and 

11 determine a series of values for one or more regions of 

12 interest, the series of values for each of the one or more regions 

13 comprising a value for each of the images. 

1 20. The computer program of claim 19, further comprising 

2 instructions to calculate from each series of values for the one or 

3 more regions a figure of merit for a library member at the 

4 corresponding library location. 
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1 21. The computer program of claim 20, further comprising 

2 instructions to display a graphical representation of each figure of 

3 merit. 

1 22. A method for evaluating a combinatorial library including a 

2 plurality of members, the method comprising: 

3 receiving a stream of data comprising a series of images of the 

4 combinatorial library; 

5 identifying a plurality of regions of interest, each region 

6 corresponding to a location in each of the series of images and to a 

7 location in the combinatorial library; 

8 determining a series of values for one or more regions of 

9 interest, the series of values for each of the one or more regions 

10 comprising a value for each of the images; and 

11 calculating from each series of values for the one or more 

12 regions a figure of merit for the library member at the corresponding 

13 library location. 

\ 23. The method, of claim 22, wherein each region of interest 

2 corresponds to a plurality of pixels in each image and the series of 

3 values for a region of interest comprises an average value for each 

4 of the corresponding plurality of pixels in each of the images. 



1 24. The method of claim 23, 

2 series of images generated at 

3 frame per second. 

1 25. The method of claim 23, 

2 series of images generated at 

3 frames per second. 

1 26. The method of claim 23, 

2 series of images generated at 

3 frames per second. 

1 27. The method of claim 23, 

2 series of images generated at 

3 frames per second. 



wherein the stream of data comprises a 
a frequency of greater than about 1 

wherein the stream of data comprises a 
a frequency of greater than about 6 

wherein the stream of data comprises a 
a frequency of greater than about 12 

wherein the stream of data comprises a 
a frequency of greater than about 20 
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1 28. The method of claim 23, further comprising displaying a 

2 graphical representation of the figures of merit. 

1 29. The method of claim 28, wherein the graphical representation 

2 comprises a histogram. 

1 30. The method of claim 22, wherein the series of images is 

2 captured from a beginning of a combinatorial experiment to an end of 

3 a combinatorial experiment and wherein the plurality of regions of 

4 interest is identified after the series of images has been captured. 

1 31. The method of claim 30, wherein the series of images comprises 

2 a series of infrared images. 

1 32. The method of claim 31, wherein the figure of merit comprises 

2 an emittance change for a library member. 

1 33. The method of claim 31, wherein the figure of merit comprises a 

2 phase transition point for a library member. 

1 34. The method of claim 31, wherein the figure of merit comprises a 

2 thermoelectric figure of merit for a library member. 

1 35. The method of claim 22, wherein identifying a plurality of 

2 regions of interest comprises receiving a first user input 

3 identifying one or more regions of interest, each region having a 

4 user-defined shape, the method further comprising receiving a second 

5 user input defining a computation for determining a series of values 

6 for each of the one or more regions of interest, 

1 36. The method of claim 35, wherein determining a series of values 

2 for each of the one or more regions of interest comprises computing 

3 each value in each series in accordance with the second user input. 

1 37. The method of claim 36, wherein the computation is an average 

2 for a group of pixels corresponding to a region of interest. 

1 38. A computer-implemented method for evaluating a combinatorial 

2 chemical experiment, comprising: 

3 • capturing a series of images of a combinatorial library, the 
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4 library including a plurality of members, the series of images being 

5 captured from a beginning of the experiment to an end of the 

6 experiment and storing the series of images in memory; 

7 after the series of images has been captured, identifying a 

8 plurality of regions of interest, each region corresponding to a 

9 location in each of the series of images and to a location in the 

10 combinatorial library; and 

11 determining a series of values for one or more regions of 

12 interest, the series of values for each of the one or more regions 

13 comprising a value for each of the images. 

1 39. The method of claim 38, further comprising calculating from 

2 each series of values for the one or more regions a figure of merit 

3 for a library member at the corresponding library location. 

1 40. The method of claim 39, further comprising displaying a 

2 graphical representation of each figure of merit. 
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